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Abstract. We establish uniform-in-bandwidth consistency for kernel-type estimators of the differen 



tial entropy. Our proofs rely on the methods of 



Einmahl and MasonI (120051) . 
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1 Introduction - Main Results 

Let Xi , . . . , X„ be independent random replies of a random vector X G M*^ with distribution func- 
tion F(x) = P(X < x) for X G W^, with d > 1. We set here X = (Xi, . . . , X^) < x = (xi, . . . , x^) 
whenever Xi < Xj, for alH = 1, . . . , d. We assume that the distribution function F(-) has a density 
/(•) (w.r to Lebesgue measure in W^). The differential entropy of /(•) is then given by 

H{f) = - [ /(x)log(/(x))dx, (1) 

whenever this integral is meaningful, and where d x denotes Lebesgue measure in R'^. The notion 



of diff erential entropy was essentially introduced by 



ShannonI (Il948h . We refer to 



Cover and Thomas 



(l2006h (see their Chapter 8), and the references therein, for details. Because of numerous applications. 



th e problem of es t imatin g H{f) has been the subject of considerable interest in the last decades (refer 



to 



Beirlant et al. 



(119971) and the references therein). The main purpose of the present article is to 
establish consistency and to provide asymptotic confidence intervals for the entropy functional H{f), 
based on kernel-type functional estimators. 
Below, we will work under the following assumptions on /(•). 

(F.l) H{f) is properly defined by the integral ([B, in the sense that 



l^(/)l<oo; 



(2) 



(F.2) /(•) is bounded and strictly positive on W^. 
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We refer to iGyorfi and van der MeulenI (1199 lb for conditions characterizing Q in terms of /(•). To 
define our entropy estimator we define, in a first step, a kernel density estimator. Towards this aim, 
we introduce a measurable function K{-) fulfilling the conditions 



(Kl) 



(K2) 



K{-) is of bounded variation on W^; 
K{-) is right continuous on W^, i.e.. 



K{ti,...,td) = lim K{ti+ei,...,td + edy, 

eiiO,...,£diO 



(K3) 
(K4) 



||ir||oo := supxgiRd \K{x)\ =: K <oo; 
/Rd K{s)ds = 1. 



We the n make use of an Akaike-Parzen-Rosenblatt (refer to 



Akaikd (119541) . 



ParzenI (119621) and|Rosenblatt 



(11956^ ) kernel estimator of /(•), defined as follows. Given a bandwidth sequence < /i^ < 1, we 



estimate /(x) by 

n 

fn,hA^) = {nK)-'J2K{i^-Xi)/hi/''), for xG 
In a second step, given fn,hS')^^^ estimate H{f)hy setting 

Hn,huAf) = - /n,h„(x)log (/„,/,„ (x))dx, 



(3) 



(4) 



n,/3 



where An^fs := {x : fn,hu{'^) ^ (log+ ^) ^} ^^d P ^ (0) 1/4) is a specified constant. Here, we set 
log_l_(n) = log(n V e) for n G M. 

The limiting behavior of /n,/i„(-)' for ap propriate choices of the bandwidth hn, has been exten- 



sive ly investigated in the literature (refer to 



and 



Devroye and Lugosi 



Bosq and Lecoutrel (|l987h . 



Devroye and Gyorfil (|l985h 



(1200 ih ). In particular, under our assumptions, the condition that /i„ — > 



together with nhn — > oo is necessary and sufficient f or the convergence in probability of .fn,h „('X- 



f(x), independently of x £ M'^ and /(•). Recently , 



Deheuvels and Mason 



(120041) . 



Einmahl and Mason 



Deheuvelsl tom. 



(l2005|)and 



Einmahl and MasonI (120001). 



Donv and Einmahll ( 2006 ) established 



uniform consistency results for such estimators, where /i„ varies within suitably chosen intervals 
indexed by n. In the present paper we will use their methods to establish convergence results for 

Hn,h„,f3if)- 

Set 

^Hr,,h„Af) = - [ ^fn,h„{^) log (E/„,^„(x))dx. 



-n,l3 



Select a sequence of constants {6„ : n > 1} such that 6n i and n6„/(logn)^+'^'^ — > (X). Our main 
result is as follows. 



Theorem 1.1 Let K{-) satisfy (K1)-(K4), and let f{-) fulfill (F1)-(F2). Then for each (3 G (0, 1/4), 
there exists a function T(c) of c > 0, such that, for each c > satisfying < cn^^ (log n)^+^'^ < 
^n < 1 with hn i 0, we have 



limsup sup < T (c) a.s. 

n^oo cn-l(logn)l+4/3<h<6„ V{logn}'^/3 (log(l//l) Vloglog 



;n 



Let {a„ : n > 1} be a sequence of constants such that < a„ < &„ < 1, together with non/ (log n)^"'"''^ 
oo. An apphcation of Theorem 11.11 shows that, with probability 1, 



\u (f\ wfT (f^\ ni ;{logri}4/3(log(l/a„)Vloglogn) 



This, in turn, implies that 

lim sup \Hn,hAf) -^^n,hAf)\ =^ «-^- (5) 

Thus we have the following corollary of Theorem ll.il 

Corollary 1.2 Let K{-) satisfy (K1)-(K4), and let /(•) be a uniformly Lipschitz continuous, and 
strictly positive density, on W^, fulfilling (Fl). Then for any f3 G (0,1/4), and for each pair of 
sequences < a„ < 6„ < 1 with 6„ | and na„/(log n)^~^^^ -^ oo, we have 

sup \K,,hAf)-H{f)\^0. (6) 

an<h<b„ 

We note that the main problem in using entropy estimates such as (01) is to choose properly /i„. The 
uniform in bandwidth consistency result given in ^ shows that any choice of h between a„ and 5„ 
ensures the consistency oi Hn,h,i3{f)- 

2 Proofs 

Proof of Theorem ll.il We first decompose H^^h^Af) ~ lE-ffn,/i„,/3(/) into the sum of two compo- 
nents, by writing 

^n,h„,/3(/)-Ei/„,;,„,^(/) = -/ /„,,„(X) log (/„,;,„ (X))dx + 

+ j E/„,/,Jx)log(E/„,ftJx))(ix 

|log/n,fe„(x) - logE/„^;,„(x)| E/„^,j„(x)dx 

|/n,/i„(x) - E/„,/i„(x)| log/„,ft„(x)dx 
^l,n,hn,f3 + ^2,n,h„,f3- (7) 



1-n,/3 



n,/3 



We observe that for all z > 0, llogz] < |i — l| + |z — 1|. Therefore, we get 

/n,/i„(x) 



log fn,h„ (x) - log E/„,h„ (x) 



log 



E/n,/.„(x) 



< 



+ 



fn,h,A^) 



IE/„,h„ (x) - f„,h„ (x) fn,h„ (x) - ^fn,h„ (x) 



Recalling that An^p := {x : fn,h„{^) > (logn)"'^}, we readily obtain from these relations that, for 
any x G An,(j, 

2 



log fn,h„ (x) - log Efn^hn (x) | < 



/n,/i„(x) -£/„,,,„ (x) 



(log n) ^ 
We can therefore write, for any n > 1, the inequalities 

|Al,n,/x„,/3| = / |log/n,/i„(x) -logE/„,/,„(x)|E/„,/,„(x)(ix 

log/„,/i„(x) - logE/„^ft^(x) E/„_/i,^(x)dx 

/n,/i„(x) -E/„,/,„(x) E/„_ft,„(x)dx 



< 



< 



■nj3 



n,p 



< 



< 



(k 


)gre)- 
2 


-/3 


(Ic 


)gn)- 
2 


-/3 



n,/3 

sup 



E/n,/i„(x) -/„,h„(x) / E/„,ft„(x)dx 



sup 



(logn) /^^gjjd 



E/nA.(x)-/n,h„(x) / E/„,;,Jx)dx. 



We note that 



E/n,/i„(x)(ix- / /(x)(ix 
By combining dH) with Theorem 9.1, page 79, in 



< / |E/„,;,„(x)-/(x)|dx. 



(8) 



Devroye and Lugosil (120011) . we obtain that 



lim / Efn,h{x.)dx = / /(x 



)dx = 1. 

Since /i„ < 6„ and 6„ | as n ^ cxd, there exists a positive constant Ci such that for all n sufficiently 
large 



< — ^ — - sup 



E/n,/i„W - fn,hni^) 



(9) 



'"^'"''^-^'-(lognH;;^-. 
We next evaluate the second term A2,n,/i„,/3 in the right side of (O. Since |log z\ < - + z, for all 



z > 0, we see that 

|A2,n,h„,/3| 



{/n,/i„(x) - E/„,,,„(x)| log/„,ft„(x)cix 

n,/3 



< / I /„,/.„ (x)-E/„,,,„(x) 



fn,hS^) 



+ fn,h„{^) 



dx. 



Similarly as above, we get, for any x G ^n,/3. 

J -- + /n,h„(x) = 

We can therefore write, for any n > 1, 

1 



1 



+ 1 /n,^Jx) 



< 



Jn,h„\^) Jn,h„\^) 

+ !)/„,;,„ (x). 



(log n) 2^ 



^2,n,h„,P\ 



< 



< 



< 



(log 


n)- 
1 


-2/3 


(log 


1 


-2/3 



+ 1) / |E/„,;,„(x) -/„,;,„ (x) I /„,;,„ (x)(ix 



+ 1 ) sup 



Observe that 



/, 



(logn) ^^ ^xGA„,^ 



n,/i„(x)dx- / /(x)(i> 



IE/n,/i„(x) -/„,/!„ (x) / /„,?,„ (x)dx 
IE/n,/i„(x) -/„,h„(x) / /„,,,„ (x)dx. 



< / |/n,h„(x)-/(x)|dx. 



(10) 



By combining (ITOl ) with Theorem 3.1, page 30, in iDevroyd (Il987h . we conclude that, almost surely 



/ /n,/i„(x)(ix ^ / /(x)dx = 1 as /i„ ^ and n/i„ ^ oo. 

Since cn~^ logn < hn < bn and 6„ | as n — > oo, there exists a positive constant C2 such that, 
almost surely for n sufficiently large, 

1 



I A. 



2,n,/in,/3 



(log n) 



-2/3 



-Fl C2 sup 



IE/n,h.„(x) -/„,/!„ (x) 



(11) 



We now impose some slightly more general assumptions on the kernel K{-) than that of Theorem 
11.11 Consider the class of functions 



/C = {i^((x - ■)/h^/'^) : /i > 0, X G E'^}. 



For e > 0, set N{e,IC) = supq N{Ke,IC,dQ), where the supremum is taken over all probability 
measures Q on (M.'^, B). Here, dg denotes the L2{Q)-metnc and N^ke, JC, dq) is the minimal number 
of balls {g : dQ{g,g') < e} of dg-radius e needed to cover /C. We assume that /C satisfies the 
following uniform entropy condition. 



(K.5) for some C > and i/ > 0, 



N{e,IC) <C£-'',0<e< 1. 



(12) 



Finally, to avoid using outer probability measures in all of statements, we impose the following mea- 
surability assumption. 



(K6) /C is a pointwise measurable class, that is, there exists a countable subclass /Cq of K, such that 
we can find for any function g{-) G /C a sequence of functions {gm{-) ■ m > 1} in /Cq for 
which 

gm{2) -^ g{z), z G W^. 



Remark that Condition (K.5) is satisfied whenever K 
is satis fied whenever i^(-) is right continuous (refer to 
J2OO5L 



is of bounded variat i on, an d C ondition (K.6) 



Deheuvels and MasonI (|2004l) and 



Einmahl and Mason 



By Theorem 1 of Einmahl and MasonI (120051) . whenever K{-) is mesurable and satisfies (K3)- 
(K6), and when /(•) is bounded, we have for each c > 0, and for a suitable function S(c), with 
probability 1, 



lim sup sup 



nh\\fn,h-'^fn, 



h. ||oo 



n^oo cn-llogn<h<l ^/log{l/h) V log log n 

which implies, in view of (O and (fTTI) . that, with probability 1, 



S(c) 



nh\Ai^ri,h,p\ 



and 



lim sup sup — -j^=^ 

n^oo cn-i(logn)l+4/3<h<6„ V {log n}^/^ (log(l//l) V log log 

nh\I\2,n,h,fi\ 



(13) 



(14) 



n) 



lim sup 



sup 



< T(c). 



(15) 

D 



i(logn)i+4/5<?t<fe„ v^{logn}4/5 (log(l//i) V log log n 
Recalling ([7]), the proof of Theorem (11.11 ) is completed by combining (IT4l) with (fTSl) . 
Proof of Corollary 11.21 Recall An^p = {x : fn,hn{'^) ^ (log+?^)~'^} and let A^ ^ the complement 
of An^fs in W^ (i.e. yl^ ^ = {x : fn,h„{^) < (log_|_ n)^'^}). We repeat the arguments above with the 
formal change of ^n,/i„./3(/) by H{f). We show that there exists positive constants Di and D2 such 
that, for all n sufficiently large. 



|E^n,/.„,/3(/)-^(/)| < 



/(x)log(/(x))dx 



n,/3 



+ 



+ 



(logn)"^'' 



+ 1]D2 sup 



Ul 


Ej 


(lognj-Z^^glRd 


IE/n,/.„(x)-/( 


x) 



IE/n,h„(x)-/(x) 



(16) 



We know (see, e.g. 



Einmahl and MasonI (120050 ). that when the density /(•) is uniformly Lipschitz 



and continuous, we have for each On < /i < 6n, as n — > 00, 



|EA,,(x)-/(x)||oo = 0(6y"). 



Thus, we have 



lim sup (logn)2/^||E/„,;,(x)-/(x) 



This when combined with ( |T6l ). entails that, as n — > oo, 

sup \\EHn,hAf)-H{f)\\^0. (17) 

a„<h<b„ 

Using (ITT] ) in connection with (|5]l imply (O. D 
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